-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HADOOP-17024. ListStatus on ViewFS root should list the linkFallBack root #2010
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make it private as we don't need to access this out side.
I think we may not need this at all. You can setConf and getConf.
And look at next comments, we can completely avoid if we use targetFileSystem instead of FileSystem.get(...conf)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use theInternalDir.getFallbackLink().targetFileSystem instead of initializing/getting another fs.
Now conf is not need at all and you dont need above uri also.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
theInternalDir.getFallbackLink().targetFileSystem is going to return ChRootedFileSystem which will wrap the path with fullPath(Path p) method. That was causing error. I have used the underlying filesystem object of the ChRootedFileSystem
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why you might have issue was because you are passing fallBackUri. ChrootedFS already contains /fallBackDir and you are again passing a uri which contains /fallbackDir. SO it will fail with "FNFE". If you use chrootedfs, you should just pass relative path by cutting chrooteduri. Since we support only at root, it will be simply "/" on chorootedfs.
I have one another concern that we are returning absolute path with fall back in ls. But with regular mountlinks we will return relative to viewfs scheme ex: [viewfs://default/data, viewfs://default/targetRoot, viewfs://default/danglingLink, [viewfs://default/user2, viewfs://default/user, viewfs://default/internalDir, viewfs://default/linkToAFile]
Since fallback also part of viewfs and its actually a child fs, should we return fallback paths also relatively constructed with respective to viewfs:// scheme?
exmaple: your fallback path is hdfs://nn1/fallBackDir and you have a dir under it. That is /user1
So, when you do ls, should it return viewfs://default/user1 ?
if you use this ls result path to create some directory like viewfs://default/user1/test1, it will/should work ( IMO) and it should create at physical location of hdfs://nn1/fallBackDir/user1/test1
Because you did not have any other mount link with /user1, it will fallback to create at fallback link.
But with the current behavior, we are returning hdfs://nn1/fallBackDir/user1 and if you use this path to create a dir under it, it will actually go to regular hdfs instance but not via viewfs. This could be a potentially problem in that case overloadScheme impl. There hdfs can go to ViewFSOverloadScheme and try to create path as hdfs://nn1/fallBackDir/user1. Now fallBackDir/user1 will be attempted to create in viewfsOS env, it checks whether we have any link with fallBackDir. Of course we will not have any regular link. So, it will create at fallback link.
Our fallback link is already hdfs://nn1/fallbackDir and now it will attempt to create /fallbackDir/user1
So, it will result to create hdfs://nn1/fallbackDir/fallbackDir/user1. This seems not correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I totally agree with your thought. Changed the path back to viewfs. Also in the unit test, verified the creation of directory with the returned path.
...ct/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
Outdated
Show resolved
Hide resolved
...ct/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
Outdated
Show resolved
Hide resolved
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
Outdated
Show resolved
Hide resolved
...ct/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may want to add the similar changes in ViewFs.java to cover FileContext based fs impl.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added change in ViewFs.java
|
💔 -1 overall
This message was automatically generated. |
a3130f4 to
2a610f9
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why you might have issue was because you are passing fallBackUri. ChrootedFS already contains /fallBackDir and you are again passing a uri which contains /fallbackDir. SO it will fail with "FNFE". If you use chrootedfs, you should just pass relative path by cutting chrooteduri. Since we support only at root, it will be simply "/" on chorootedfs.
I have one another concern that we are returning absolute path with fall back in ls. But with regular mountlinks we will return relative to viewfs scheme ex: [viewfs://default/data, viewfs://default/targetRoot, viewfs://default/danglingLink, [viewfs://default/user2, viewfs://default/user, viewfs://default/internalDir, viewfs://default/linkToAFile]
Since fallback also part of viewfs and its actually a child fs, should we return fallback paths also relatively constructed with respective to viewfs:// scheme?
exmaple: your fallback path is hdfs://nn1/fallBackDir and you have a dir under it. That is /user1
So, when you do ls, should it return viewfs://default/user1 ?
if you use this ls result path to create some directory like viewfs://default/user1/test1, it will/should work ( IMO) and it should create at physical location of hdfs://nn1/fallBackDir/user1/test1
Because you did not have any other mount link with /user1, it will fallback to create at fallback link.
But with the current behavior, we are returning hdfs://nn1/fallBackDir/user1 and if you use this path to create a dir under it, it will actually go to regular hdfs instance but not via viewfs. This could be a potentially problem in that case overloadScheme impl. There hdfs can go to ViewFSOverloadScheme and try to create path as hdfs://nn1/fallBackDir/user1. Now fallBackDir/user1 will be attempted to create in viewfsOS env, it checks whether we have any link with fallBackDir. Of course we will not have any regular link. So, it will create at fallback link.
Our fallback link is already hdfs://nn1/fallbackDir and now it will attempt to create /fallbackDir/user1
So, it will result to create hdfs://nn1/fallbackDir/fallbackDir/user1. This seems not correct.
| FileStatus[] mountPointStatuses) { | ||
| ArrayList<FileStatus> result = new ArrayList<>(); | ||
| Set<String> pathSet = new HashSet<>(); | ||
| int i = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove above unused variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
💔 -1 overall
This message was automatically generated. |
...p-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
Show resolved
Hide resolved
|
Could you also check CI results? Seems like checkstyle issue related to this change? Thank you. |
|
Thank you @abhishekdas99 for addressing all the comments. Nice work! |
Thanks @umamaheswararao for your guidance. |
|
💔 -1 overall
This message was automatically generated. |
|
Thank you @abhishekdas99 for the contribution. I have just committed this to trunk. |
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).